The duration problem with multiple 

exchanges ^ 



Charles E.M. Pearce^ 

'The University of Adelaide, School of Mathematical Sciences, Adelaide, Australia 

SA 5005 



Krzysztof Szajowski 



^Wroclaw University of Tech., Institute of Mathematics and Computer Science, 
Wybrzeze Wyspianskiego 27, 50-370 Wroclaw, Poland 

Mitsushi Tamaki ^ 

^Aichi Univ., Nagoya Campus: 370 Kurozasa, Miyoshi, Nishikamo, Aichi 470-02, 

Japan 



Abstract 

We treat a version of the multiple-choice secretary problem called the multiple- 
choice duration problem, in which the objective is to maximize the time of possession 
of relatively best objects. It is shown that, for the m-choice duration problem, 
there exists a sequence (si, S2, • • • , Sm) of critical numbers such that, whenever there 
remain k choices yet to be made, then the optimal strategy immediately selects a 
relatively best object if it appears at or after time Sk {1 < k < m). We also 
exhibit an equivalence between the duration problem and the classical best-choice 
secretary problem. A simple recursive formula is given for calculating the critical 
numbers when the number of objects tends to infinity. Extensions are made to 
models involving an acquisition or replacement cost. 
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1 Introduction and summary 



Ferguson et al.l (119921 ) were the first to consider a sequential and selection problem referred 
to as the duration problem, a variation of th e classical secret ary p roblem a s treat ed by 
Gilbert and Mostellerl (1l966l ) and others (see iFergusonl (119891 ) and ISamueld (1l99ll ) for a 
history and review of the secretary problem). The basic form of the duration problem in 
the no-information setting can be described as follows: a set of n rankable objects appears 
one at a time in random order with all n\ permutations equally likely. As each objects 
appears, we decide either to select or reject it based on the relative ranks of the objects. 
The payoff is the length of time we are in possession of a relatively best object that have 
appeared to date. Thus we will select only a relatively best object, receiving unit payoff 
as we do so and an additional unit for each new observation, as long as the selected object 
remains relatively best. 



Though jPerguson et al.l (119921 ) considered the various models extensively, they confined 
themselves to the study of the one-choice problem. We consider here, as a natural general- 
ization of their study, the no-information mult iple-choice du r ation problem and its mod- 
ifications. Preliminary results are included to iTamaki et al.l (119981 ). The multiple-choice 
duration problem is reformulated as the multiple optimal stopping problem, which has 
been treated by many authors. The double-stopping problem was pos ed by HaggstronJ 
(11967) a nd for discrete-time M arko v proce s ses ha s been considered by lEhj dukyavichyus 



Nikolaev 


(1979. 


1998) and 


Stadje ( 


1985) 



For the m-choice duration problem, we choose at most m objects sequentially, and receive 
unit payoff at each time point as long as the last chosen object remains a candidate, that is, 
a relatively best object. Only candidates can be chosen, the objective being to maximize 
the expected total payoff. More formally, this problem can be described as follows: let 
Tn{i) denote the arrival time of the first candidate after time i if there is one, and n + 1 



if there is none, so 7^(2) : f2 — -|- 1, 



n 



1}. Then 



is the duration of the candidate selected at time i and the objective is to find a stopping 
vector (rj*, r2 , . . . , r^) such that 



E 



i=l 



sup E 

("""l ''''2 ' ■ ■ ■ ''''m ) SCm 



i=l 



Here Tj (1 < z < m) denotes the stopping time related to the z-th choice and Cm is the 
set of all possible vectors (ri, r2, . . . , r^). 



A generalization of this problem is considered in Section [231 which we allow the number 
M of objects presented to be a random variable. We show for the m-choice duration 
problem that, subject to a condition on the distribution of M, there exists a nonincreasing 
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sequence (si, S2, • • • , Sm) of critical positive integers such that, whenever there remain k 
choices to be made, the optimal strategy immediately selects a candidate if it appears 
at or after time Sk (1 < < m). That is, the optimal strategy has a threshold form. 
In Section 12.31 we show that this condition is satisfied in the two particular cases when 
M ~ Ms{n) (the distribution with P{M = n} = 1 i.e. the degenerate distribution) and 
M ~ Mu{n) (the distribution with P{M = i} = 1/n {i = l,...,n i.e. the discrete 
uniform distribution). 



In Section 12.41 we investigate the asymptotics for n — » oo in the case when M has the 
degenrate distribution P{M = n} = 1. The ratio Sk/n converges to a limit si G (0, 1). A 
recursive formula for calculating in terms of si, Sg, . . . , sl_i is given by 



exp 



\ 



fe-i 



1-2E 



[{k - i + 2) + {k - i + l)\ns* 
{k-i + 2)\ 



■(In s 



*\k-i+l 



(1) 



Throughout the paper the empty sum is taken as zero, so the above formula is valid for 
all /c > 1. We show also that, as n — oo, the expected proportional payoff, that is, the 

m 

expected maximum payoff per unit time, is given by -J2 ■s^lns^. 

fc=i 

The classical best-choice secretary probler n (BCSP ) is co ncerne d with maximiz i ng the 
probability of choosing the best object. I Samuels! (jl99ll ) and iFerguson et al.l (119921 ) 
pointed out that, for the one-choice problem, the duration problem with a known number 
n of objects is equivalent to the BCSP with an unknown nur nber of objects havin g a un i- 
form dis tribution on |1, 2 , . . . , n}. This was fir st studied by iPresman and SoninI (119721 ). 
See also iPetruccellil (|l983l) and iLehtinenI (Il993l) for th e probl em wi th an un k nown number 
of objects. Icilbert and Mostellerl (1966), Sakaguchi ( 1978 ) and Preater ( 1994 ) for the 



multiple-choice problem and 



Tamaki 



"(Il979bh 



for the formulation of the multiple-choice 



problem with an unknown number of objects and solution of the two-choice problem 
having a uniform prior of the actual number of objects. 



We show in Secti o n 12.31 that this equivalence still holds for the multiple-choice problem. 
Recently iGnedinI (120051 ) established an equivalence between the various best-choice pro b- 
lems and the re l ated duration problems in a greater generality (see also iGnedinI (12004] )). 
Ferguson et al.l (119921 ) considered another type of problem, the full-information duration 
problem, where the observations are the actual values of the objects assumed to be inde- 
pendent and identically distributed (iid) from a known distribution and hence decisions 
are based on the actual values of the objects. They showed that the above equivalence be- 
tween the best-choice proble m and the duration problem holds also for the full-information 
problem. iPorosihskil (119871 ): iPorosinskil (120021) consider the full-informati on be st-choice 
proble m with a random number of objects and iMazalov and Tamakil (120061 ) and I Samuels 
(120041 ) the limiting maximum proportional payoff for the full-information one-choice du- 
ration problem. 



In Sections [3] and m the multiple-choice duration problem with M ~ Ms{n) is generalized 
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by introducing costs. In Section [3] a constant acquisition cost is incurred each time an 
object is chosen, while in Section H] a constant replacement cost is incurred with the 
selection of any candidate other than the first. The objective in Sections [3] and H] is 
to maximize the expected net payoff. It can be shown that, under an appropriate cost 
condition, the optimal strategies have similar structure to that for the problem involving 
no cost. In Sections 13.21 and we investigate the respective associated asymptotics. 



The multiple-choice duration problem with replacement and acquisition costs may be con- 
sidered a s a marriage and divorce problem , interpreting the replacement cost as alimony. 
Recently. ISeale and RapoportI (119971 . 120001 ) investigated the behaviour of the decision mak- 
ers under circumstances similar to those of the best choice pro blem model . The discussion 
of the problem are the subject of papers by lBeardenl (120061 ) and lSzajowskil (120061 ). It seems 
important to construct a model with physical parameter to fit it to the empirical data. 
The considerations of sections [3] and H] suggest one way forward. 



2 Multiple exchange and hold of the relatively best item 



In this section we address the optimal choice problem with an unknown (bounded) num- 
ber of objects. At most n objects appear in turn before us. We have only an a priori 
distribution pi = P{M = i} on the actual number of objects, where J27=iPi = 1- Without 
loss of generality, we may assume that p„ > 0, so tTj > for 1 < i < n. We set J2]=iPi = T^i- 
We are allowed to make at most m choices and wish to maximize the expected duration 
of holding a relatively best object. 

We assume that all that can be observed are the relative ranks of the objects as they 
are presented. Thus if Xi denotes the relative rank of the i-th object amongst those 
observed so far (a candidate if = 1), the sequentially-observed random variables are 
Xi, X2, . . . , Xn- It is well known that under the assumption that the objects are in random 
order with all n\ permutations equally likely, we have 

(a) the Xi are independent random variables and 

(b) P{X, = j}=l/z, for 1 < J < z, 1 < z < n. 

We formulate the m-choice duration problem as a Markovian decision-process model. First 
we condition on M = I > i. Since decisions about selection or rejection occur only when 
a candidate appears, we describe the state of the decision process as (i, /c), 1 < i < £, 
1 < A; < m, if the i-th object is a candidate and there remain k more choices to be made. 
For the above process to be a Markov chain, we must further introduce an additional 
absorbing state {I + l,k,e) for the situation where the last object is presented at time i 
and is not a candidate, with k choices left (1 < A; < m). When it leaves [i, k), the process 
moves to a state (j, k — 1) if the i-th object is selected. Otherwise it moves to a state (j, k) 
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or {i + l,k,e). By (a) and (b), the distribution of j is given by 

j = i + l, 



We now remove the conditioning on M. The probabihty of a next candidate appearing at 
time j {i < j < n) is given by 

P(*,J) = ^^--- (3) 

The corresponding probabihty of transition from (z, k) to a state (j, /c), (j, k — 1) or (j, fc, e) 
is 

In accordance with our convention about empty sums, we shaU interpret 7r„+i as zero. 
Let / : {1, 2, . . . , n + 1} — s> 3? be the payoff function. Define 

n+l 

T/(0 = E /(J)P(^, j) (5) 
the expectation operator with respect of the probabihty distribution ([3]) and 

n+l 

T7(^) = E fiMh j) (6) 
i=i+l 

the expectation operator with respect of the probabihty distribution (jl]). 

If the z-th object observed is a candidate, the period for which it remains a candidate has 
mean 



where 



^=1 



When TTj = 1, for j = 1,2, ... ,n then the sequence Lj given in ([8]) is denoted by Hj (see 
section [5] for further details and section [231 for properties given by formulae ( 13T|) ). The 
contribution to the expected occupancy time from a further candidate, if any, is 



r(l) ^ 1 



K^'^ = T[4:;.)] = E = - E -31 - ^^-0 • (9) 

j=i+l * j=i+l 
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When n < m, the optimal strategy is easily seen to be to select the candidates successively 
as they appear. Thus we assume n > m. Before proceeding to investigate the optimal 
strategy, we introduce some notation. Suppose we start in state (z, k). We denote by t/f'^'*, 
V^^^ the expected total possession time when we select or reject respectively the i-th 
object and then proceed in an optimal manner. We also denote by W^/^^ the expected 
total possession time under an optimal strategy starting from state {i, k) [1 < i < n, 
1 < k < m). The Bellman principle of optimality yields for 1 < i < n and 1 < k < m 
that 

Wt^=max{ut\vl'^}, (10) 

n 

= ^^'^+ E pihj)wi'-'\ (11) 
j=i+i 

n 

j=i+i 

Equations (fT0l) - (fT2l) together with the boundary conditions Wj^^'^ = for 1 < i < n + 1 

(k) 

and W;C+i = for 1 < A; < m can be solved recursively to yield the optimal strategy and 
the optimal value Wi^\ When n < m selecting the candidates successively as they appear 
is the optimal strategy and achieves total possession time n for relatively best objects. 
Thus we may assume n > m without loss of generality. 



2.1 The main auxiliary theorem 



In the sequel, in the construction of optimal solutions, the properties of the sequence 
of differences between some payoffs and expected payoffs will be analysed. Important 
properties of such sequences are gathered in the following theorem. 

Theorem 2.1 Suppose that si and N are integers with 1 < Si < N . Suppose further that 
(GP)^i is such that 

(CI) z/Gf^ > with l<i<N, then G'S+\ > 0; 
(C2) z/ 1 < z < si, then G\% > G\^\ 
(C3) Si is determined by 

si = mm{i : ^ > 0}. 

Then if 1 < m < N there exists a sequence (si, S2, . . . , Sm) of positive integers such that 
if, for 1 < i < N , we define 

N 1 

Gf)=Gf^+ y —L^Gf^i) (k>2) (13) 

— ^ 7 — 1 

j=max{i+l,Sfe_i) 

recursively for 1 < k < m, then we have the properties: 



6 



(Pl)fc z/Cf^ > with l<i<N, then g\% > 0; 
{P2)k ifl<k<m, then Cf^^^ > Cf^; 
{P3)k in<i< Sk, then > Cf^; 
{P4:)k Sk is determined by 

Sk = mm{i : ^ > 0}; (14) 

(P5)fc (si, . . . , Sk+i) is nonincreasing for 1 < k < m. 

Proof. We shall employ induction on k. From (C1)-(C3) we have (-Pl)i, (-P3)i and (-P4)i 
and that G) is well-defined for 1 < z < A^, with 



G? = G^^+ E (15) 



By (-Pl)i and (-P4)i the summand in (ITS!) is nonnegative so that (fT5|) yields (-P2)i. From 
this and (-P4)i we deduce (-P5)i. Thus we have a basis for the induction. 

For the inductive step, suppose that for some k with 1 < k < m {Pl)£-{P5)e hold for 
1 < i < k. By definition we have for 1 < z < that 

.(fc+2) ^(fe+1) _ J- ^(fe+1) i ^(k) 



j=max(i+l,Sfe+i) j=max(i+l,Sfe) 

From {Pl)k and (-P4)fc the second summand is nonnegative. Since Sk > s^+i, we thus have 



N 1 



- > 2^ — Y ^Gj- - 

j=max(i+l,Sfc+i) 

Hence {P2)k implies that 

and we have {P2)k+i. Thus is well-defined by (IT^ and (P4)fc+i applies. Also < 
Sfc+i, so (P5)fc+i holds. 

Since < si, (P3)i implies G^+i > ■* for 1 < i < — 1. For any such values of i, 

N 1 

^(fe+1) _ ^(1) V- 

j=Sk ^ 

j = Sk ^ ^ 

Thus 

cStt'^ > for 1 < z < Sfe - 1. (16) 

Also by ([HD we have for i = Sk+i - 1 that Cf^^^^ < and Cj+t^^ > 0. Hence (P3)fc+i 
apphes. 
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This leaves {Pl)k+i- Since Gg+i) > 0, we have by (\Wi) that > for Sk+i < i < Sk- 

Further by {P2)k we have Cf "^^^ > for z > Sk- Hence Cf > for z > Sfc+i and 
Gf < for ^ < Sfc +1, giving (Pl)fc+i. This completes the inductive step. 



2.2 The basic theorem 



We complement relations (fTU]) - (fT^ with V}''\ the expected total possession time if a 
candidate at time i is rejected and the next candidate (if any) accepted, with optimal 
choices following such acceptance. 

Theorem 2.2 For 1 < i < n, let be given by below with k = 1. Suppose that for 
N = n and some integer Si, conditions (C1)-(C3) of Theorem \2.1\ are satisfied. Then for 
the m-choice duration problem with k (1 < k < m) choices still to be made, the optimal 
strategy selects immediately the first candidate, if any, to appear at or after time Sk of 
Theorem \2.1\ 

Proof. Since conditions (C1)-(C3) of Theorem 12.11 are satisfied, the conclusions of The- 
orem 12.11 hold. We proceed inductively, establishing the following: 

{Ql)k the optimal strategy when there are k choices still to be made to to select the first 

candidate to appear at or after time Sk] 
{Q2)k for 1 < z < n 

111 [c/.C^) _ -(/.C^)! - nif") 



or. (17) 



For a basis, consider the one-choice duration problem. By definition (Q2)i is given. We 
should select a candidate observed at time i in preference to the next candidate (if any) 
iiU^'^^ > Vl'^\ that is, if 

^ > 0. (18) 

By (-P4)i of Theorem 12.11 this condition cannot be satisfied if i < s^. Also by (-Pl)i and 
(P4)i, the first candidate at or after time Si satisfies (JTSj) and choice of this candidate is 
strictly preferable to choice of any candidate subsequent to the second candidate after Sk- 
Thus {Ql)i holds and k = 1 provides a basis for induction. 

For the inductive step, suppose {Ql)e and {Q2)i to be true for £ = 1, . . . , A; — 1 for some 
k with 2 < k < m. Then 

I tVj , J ^ Sk-l 
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and 



By the Bellman principle of optimality we have 



(20) 



.(1) , TAk-iy 



j=i+i 



j=i+i 



(fc-i) 



by©. 

By ([19]), subtraction of from ([U]) yields 



(21) 



j=max(i+l,Sfe_i) 
n 

i=max{j+l,Sfe_i) 



..(fc-l) _ T.(fe-1) 



(by ( l20i) ). Thus the inductive assumption provides 



i 



Jjik) _ y (k) 



The recursive definition of the functions G) leads to 



Gi'' + E 

j=max(«+l,Sfc_i) 
(k) 



_J_Qik-l) 



vr,- 



G 



and we have established {Q2)k. The argument leading from {Q2)k to {Q\.)k follows that 
leading from {Q2)i to {Ql)i and the inductive step is complete. 



2. 3 Applications of the basic theorem 



We can apply Theorem 12.21 whenever we can verify conditions (C1)-(C3). To this end, 
we note that by ([7]) and ^ 



G\ ^ — Ln — Li^i — E ~ Ti-^n — Lj^l) 

j=i+l J - J- 

for 1 < i < n, whence we derive that 



(22) 



Gi+i ^ G^^^ according as Li + tTj 



(23) 



for 1 < i < n. 
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Proposition 2.3 For 1 < i < n put ipi = {i + l)vrj. Then a sufficient condition for 
(C1)-(C3) to hold is that there should exist an integer i^ with 1 < io ^ n such that 

i'i<i'2< ■■■<^io and V'io > ^io+i > • • • > ^n- (24) 

Proof. Since < (j)n and G^^-* = 7r„/?T, > 0, we have readily from fl23|) tliat if 1 < io < n 
witli 

(pi < 4>2 < ■ ■ ■ < 4>io and (pig > >...></)„, 
then g\^^ satisfies (C1)-(C3). Also ^ 0j according as ^ The stated 

result follows. 



As corollaries we consider the two special choices M ~ Ms{n) (when M = n with proba- 
bility one) and M ~ Mu{n) (when M is uniformly distributed on {1,2, ... ,n}). 

Corollary 2.1 For 1 < m < n, the optimal strategy in the m-choice duration problem 
with M ~ Ms{n) is given by the conclusion of Theorem \2.2[ 



Proof. Here tt,/ = 1 for 1 < i < n, so (1241) holds with io = n. The stated result follows 
from Theorem 12.21 and Proposition 12.31 



Corollary 2.2 For 1 < m < n and M ~ Mu{n), the optimal strategy in the m-choice 
duration problem is given by the conclusion of Theorem \2.^ 

Proof. Here Hi = {n — i + l)/n for 1 < i < n, so ipi = (i + l){n — i + 1) /n and 

ijji+i ^ ^}Ji according as n ^ 2i + 1. 

Hence (l24l) holds with io = max{l, [(n— 1)/2J } and the stated result follows from Theorem 
[2T2] and Proposition [273! 



For m > 1, the expected payoff for the m-choice problem with M ~ Ms{n) is qm = Wi 
We have 



(m) 



" j = Sm J ^ t=j ^ j=S^ JU '-J 

The Vl'""^^ (m > 2) may be calculated recursively from 



Tr(m-1) _ I Qm-l, i < Sm-1 — 1 



with the interpretation that = 0. 
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In the three subsequent sections on asymptotics it is convenient to scale mean durations 
by dividing by n so as to work in terms of the average possession times per unit time. 
We shall set uf^ = Jjf'^ jn, etc. This leaves optimal strategies unaffected. As a prelude 
to this, we consider one further application. 

We compare two differently formulated multiple optimal stopping problems. The first is 
the m-choice problem of Corollary 1 2 . 1 1 and the second the m-choice best- secretary problem 
with an unknown number of objects having distribution Muin). We show that these 
problems have the same solution in the sense that the optimal strategies and expected 
payoffs are the same. In the latter problem we win if the last chosen object is best overall. 
The objective is to maximize the winning probability. 



Tamakil (jl979bl ) formulated this problem as a Markovian decision process model and 
solved explicitly the two-choice problem with a uniform prior on M. We describe the 
state of the process be described as (i, /c) (1 < i < n, 1 < /c < m) if the i-th object is 
a candidate and there remain k choices to be made. We denote by ui^{vi'^^ the winning 
probability when we select (reject) the i-th object and then continue optimally from state 
(i, /c). If we let 



(fc) r (fc) {ky 

w\ = max i u) 



\uf\vf^}, l<^<n (27) 
the principle of optimality yields 



4'>=i:-^+ 1 -r^^-r". (28) 



n 



When M ~ M„(n), we have tt, = {n — i + l)/n for i = 1, . . . , n. If we set 

Uf' = '^^^v!t\ Ff) = ^^^^.P and wP = '^^^wf\ (30) 
n n n 

(17n) - (12Ul) are transformed respectively into 



lyf =max{t/f\v^f^}, 

These are ffTU]) - f[T^ for the scaled version of the process of Corollary l2.lt with the correct 
normalized value for . Because of the common multiplicative factors in f[30|) . i/;) = 
if and only if wj^'^'^ = uf'\ Thus optimal choices are the same in the two processes. Since 
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(k) (k) 

Wi = wi , the two also share their optimal payoff value. Thus we have established the 
following result. 

Theorem 2.4 The optimal- choice strategy and expected payoff' is the same for the m- 
choice versions of 

(i) the best-choice secretary problem with unknown number of objects distributed uniformly 
on {l,2,...,n}; 

{a) the duration problem with possession times of relatively best objects scaled by division 
by n. 



2.4 Asymptotics for the basic problem with the degenerated distribution of objects 

It is of interest to investigate the asymptotic behaviour of Sk/n [1 < k < m) and qm/n 
as n — s> 00. To do this, we observe that the sums in the formula of Section 2 are Riemann 
sums. With M ~ MJn), (ESI) becomes 



^ =H^- - -^(^n - (31) 

j=i+i 3 ^ 

for 1 < i < n, where = EL^ 1/^ for £ > 1 and Ho = (see also ([8])). 

For m = 1 with i/n ^ x as n ^ 00, the Riemann sum given by converges to the 
integral 

Jx y Jx y Jy z 2 

From ([HD, 

si = lim si/n = (33) 
is obtained as the unique root x G (0, 1) of (^^^^(a;) = 0. 

More generally (fT5]l leads to functions G^''\x) (0 < x < 1) defined recursively by 

GW(x) =G(^)(x) + -G''''-^\y)dy, k>2 (34) 

J max{a;, y 

with Gj-^'' a Riemann approximation to G^'''^{x) ifz/n— s-zasn— s-oo. 

Correspondingly si := Jim^Sfc/n exists and may be obtained for /c > 2 as the unique root 
X G (0, 4_i) of 

G^''\x) = 0. (35) 

12 



From (l34l) and (135!) . is a root of 
or equivalently, from ( l32|) . 



4-1 



Sfc = exp <^ - 1 



1 + 2/' ^5^., 



To derive the tractable form ([T]), we need some lemmata. 



Lemma 2.3 For k a positive integer, define 



J si ^ X 



1st ^ X 



<2 < A;-l. 

Si . X 



Then i satisfies the recursion 



1 



Ak,i — cik^i + —— — (In 

z + 1 L 

with Afc^fc = (k>^). 
Proof. From (1341) 



Since s^_j < s^„j„i, we have 



G^^\x)^\l\G^^~'-^\y)dy, x>s 



A 



k, i 



+ 



(In x) 



(Inx) 



■ + Ak_i_i Q 



\G^^\x) + Ak-i-i,o]dx 

G(i)(x) + C - G^^-'-^\y)dy 
Jx y 

(Inx)* , 
dx - 



dx 
y (Inx) 



-dx 



X 



The second and third terms of the last line are respectively 

Ak~i~i,Q 



(ln4_,_i)^+^-(lnsU 
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and 



1 



1 



whence the desired result. 



For simphcity, set Ak := Afc,o- Repeated use of (!38|) gives the following recursion for A^. 



Lemma 2.4 For k > 1, Ak satisfies the recursion 



i=l 



ak,k~i (Ins 



{k-i)\ {k-t + l)\ 



Ai-i 



For > 1, define A^^ by 

Nk = -{l + ^l + 2Ak.i). 
Then from fl36|) 

= exp Nk 
and we have the following lemma. 
Lemma 2.5 For k > 1, satisfies the recursion 



\ 



1=1 



{k-i + 2)\ 



(39) 



(40) 



(41) 



Proof. Straightforward calculation from fl37j) yields 



k-i-l 



k-i+2 



O^k-l, k-i-l 



k-i + 1 2{k-i + 2) 



and = Ni + Nf/2, 



so from Lemma [2.41 



fc-i 



Ak-i = E 



i=l 



O'k-l, k-i-l {Ni) 



k—i 



{k-i-l)\ {k-l)\ 



Ai^i 



^ _ {(A: - z + 2) + (A; - z + l)Ni}{Nif-'+^ 
Combining this with (1391) completes the proof. 
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By fHU]) . recursion ([T]) is an immediate consequence of dH]). From ([T]) we have successively 



s*=exp{-2} ^ 0.1353, 
$2 = exp 



S3 = exp 



S4 = exp 




V 



31 2 1^ 7 

\ 15 + 14W- 

45 81 I V 3 



0.0311. 



See Tabled] for and sIq (c = 0). 

We have the following lemma concerning the expected payoff. 



Lemma 2.6 Let = lim qm/n for m > 1. Then 



(42) 



k=l 



Proof. For m = 1, we have from fl25l) that 



^ } dy } dz 



y J z 
s*, y 



1 /I *\2 



By f l33p . this may also be written 



Qi = -Silnsi. 



(43) 



For m > 2, we have from fl25l) and fl26l) that 



y J z 

s* V 

m y 



-i\/(™-i)(y)rfy 



* 1 



(44) 
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if V^"^ ^•'(x) (0 < X < 1, m > 2) are defined recursively by 



T/(m-l)|' N = J 1 1 1 

I. X y X " 

starting with V^^\x) = 0. 

On the other hand, we have from ffTTl) and fl2Tl) that for k > 2 



(1) 



(fc-i) 



E 



^ T/(fc-l) 



by (fT2!) . On letting i/n ^ x as n ^ oo, we derive from the case k = m that 



Then ^('"^(s;;) = implies that 

1 } 1 

si J 



or equivalently, from ({32]) and V^^"" ^H'Sm) = 



J_y(m-i)^^-j^^ ^ (2 + In sl^) In ^ gm-l 



(45) 



Apphcation of fH5|) to (jHj) provides gj^ = — sj^lnsj!^ + which upon repetition and 

use of (iSl) provides (|l2l). 



Numerical values of the first four gj^ are g* = 0.2707, ^2 = 0.4725, ^3 = 0.6208, ql = 
0.7287. See Table [2] for g* and g*o (c = 0). 

Remark 1 The above approach is rather intuitive. To make the argument more rigorous, 
we can appro ximate the difference equations by differential equations. This method was 
suqqested by iDvnkin and Yushkevicn nl96U) and has since been app lie d successfully by 
Szajowsh (1981), Suchwalko and Szajowskl (200i ) and Yasuda (1985). Mucci (1973a I ) 
has developed the idea for a wider class of optimal stopping problems. 
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Since 



lyf ) and = E 



(k) 



we have by subtraction that 

n. 



J=l 



i - 1 



W, 



(fc) 



yik) _ ^(fc) 



^/so = f/« + r/^'-^^ so 

Wt^ = max y/'O = max (y/'^), y/'^) + U^'^ 

Hence 



(1) , T/C^-l) _ y W 



/n the development of the differential equation approach cited above it can be shown that 
asymptotically vj;'^'^ /n ^ f'^^\i/n). With i/n ^ x as n —>■ oo, we derive 



ax 



-x\tix + f^^-^\x) - f'^^\x) 



< X < 1 



with boundary condition f^^\\) = 0. Here the nonincreasing sequence of critical numbers 



is 



si: f^'\sl) = f^'-'\sl)-sllnsl, k=l,2,... . 

The function f^''\-) is constant on [0, s^], so the expected payoff is ql = /*-^^(0) = f^'^^sl). 
For example, routine algebra yields for k = 1 and 2 that 



x] 



—si \nsl, < X < si 
f (Inx)^, sl< X <1 



f''\ 



x] = { 



—si In si — ^2 In S2, < X < S2 
2sl - fx + |(ln xf, sl<x< si, 
|(lnx)^ — |(lnx)^, sl<x<l, 



where $1 = exp(— 2) and = exp{ — (1 + a/7/3)}. For k > 3, we can proceed in similar 



way. 
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3 The multiple-choice duration problem with acquisition costs 



In this section, the multiple-choice duration problem is generalized by imposing a constant 
acquisition cost c = c(n) > each time an object is chosen. The objective of this problem is 
to maximize the expected net payoff, that is, total possession time less the total acquisition 
cost incurred. 



3.1 The degenerate distribution of the number of objects 



For simplicity we restrict attention to the case P{M = n} = 1, so that vTj = 1 for 
1 < i < n. To avoid triviality we assume n> 1. 

Consider first the one-choice problem. The expected net payoff resulting from a choosing 
a candidate presenting at time i is 

l4" := f/i" - c, 

which by ([7]) is given by 

) = i{Hn - H,_,) - c. (46) 
We have that 

Ui\i — = Hn — Hi — 1, 
which is strictly decreasing in i and is negative for i = n — 1. 

Put 

K{n) := min{i ■.H^-Hi<l}. (47) 
Then 

Oil ^1^2 S • • • S I^K{n) ailCL lAK{n) ^ ^K{n)+1 ^ ■ ■ ■ ^ ■ 

If I^kIu) — 0' optimal never to choose a candidate, so without loss of generality we 
may assume l^^jn) > 0' ^^^^ 

< c < U^^ly (48) 

Further, there exist integers a = a{n, c), b = b{n, c) satisfying 

1 < a < K{n) <b<n 

such that U^^^ > if and only ii a < i < b and to maximize expected total payoff we 
never choose a candidate presenting at time i when a < i < b fails. Clearly this holds also 
in the m-choice problem. 
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We define vf^^ as the expected net payoff when we reject a candidate appearing at time 
i < b but select the next candidate (if any) appearing no later than time b. We then have 



(1) 



j=i+i 



j=i+i 



1 1 

I b 



for i < b. 



(49) 



We now turn attention to the m-choice problem. For i < b we employ the notation 



Vt\ Vf \ ) analogously to ut\ 



v/'^'*, wj;''\ respectively and referring to 



expected net maximal payoff rather than expected total time of possession of candidates 
and with choice of second and subsequent candidates occurring no later than time b. 
If m > 6 — a it is clearly optimal to simply choose every candidate appearing in /, so 
we suppose m < b — a. The following theorem summarizes the optimal strategy for the 
m-choice problem with acquisition cost. 

Theorem 3.1 For the m-choice duration problem with acquisition cost c subject to [JW ; 
there exists a sequence (si(c), S2(c), . . . , Sm{c)) of integral critical numbers such that, when- 
ever there remain k choices to be made, the optimal strategy selects the first candidate to 
appear at or after time Sfc(c) but no later than b. Moreover Sk{c) is nonincreasing in k 
and determined by Theorem \2.1\ with N = b and 



G 



(1) 



" 1 c 
Hn — Hi-i — E ~ (-^" ~ -^i-i) ~ ^- 



(50) 



j=i+i 



Finally, Sm{c) > a. 



Proof. There is nonnegative expected payoff from a candidate selected at time i with 
a < i < b, but not for one selected after time b, so it suffices to establish the result for 
candidates arriving at times i < b. 

From (l46l) and (1491) . we can verify that (150!) is equivalent to 



G 



(1) 



We derive that for i < b, 



1 

i 

(1) 



U 



(1) 



V, 



(1) 



1 



[Hn — Hi_i] 



The right-hand side is strictly decreasing in i for i < K{n) and nonpositive for K{n) < 
i < b. It follows that (C1)-(C3) of Theorem 12.11 are satisfied provided that G^^"* > 0. 

To see that this requirement is met, observe that Uj;^^ > 0, that is, 



Hn — Hi 



b(n)-l 



c > 0. 
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Hence 



as required. 



G^h^ = Hn — Hb-l — 7 > 



Thus the conditions of Theorem 12.11 are met. Estabhshing the theorem now follows closely 
the rest of the proof of Theorem 12. 2 ^ operating on the interval [1, b] instead of [1, n]. Since 
a candidate arriving before time a is never accepted, we have finally Sm{c) > a. 



The case c = corresponds to the duration problem treated in Corollary 12. 1[ Thus 
we have a{n, 0) = 1 and b{n, 0) = n. In Section 8 we shall need to compare quantities 
occurring in that context and the present one. Accordingly we shall where necessary for 
clarity write the Cf'^ occurring in this section as G[''^ (c) and that of Corollary 12.11 as 
G[''\o), etc. We shall need the following result. 

Corollary 3.1 For c > and i < h{n,c), the value of gi[c) = G|^^(0) — G'i^\c) is 
independent of i and so may be written g{c). Further 

g{c) > c/n. (51) 
Proof. We have for i < b that 

n 1 

g,{c) = c/6 - 5: — - {Hn - H,^,) , 
j=b+i J ^ 

which is independent of i. Also j > b implies that j{Hn — Hj^i) < c, so 

1 „ . A c /II 



■4", ? - r ^ ' .4", ?■ ? - 1 



c - — 



from which fl5T]) follows. 



3.2 Asymptotics for the duration problem with acquisition costs 

Observe first that, from ( 1471) . lim Kin) In = e~^, so the cost condition ( 1481) is reduced, 
as n — i> OO, to 

c= lim c(n) /n < . (52) 
After division by n, we may let i/n — > x as n — ^ oo in (146|) to show that U^^^ approaches 



U^^^ (x) = — c + X / — = —c — xlnx. 
Jx y 
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Let P = linib{n,c)/n. Then /3 is the unique root x E [e ^,1) of f/'-^-*(x) = under the 
cost condition fl52|) and satisfies 

-P\n(3 = c. 

For k > 1, define si = s^.(c) = luil^Sk{c) /n. When we let i/n — a; as n — oo in (l50l) 
divided by n, G^^\x) approaches the integral 



x) 



^ dy fP dy dz c (2 + Inx) In x - (2 + In /?) ln/3 



X y Jx y Jy z (3 



(53) 



Thus s\ = exp{ — (2 + In /?)} is obtained as the unique root x G (0, p) of = 0. For 

< X < P and k > 1, define G^''\x) recursively by 



= G^^\x) + / - G^''"^\y) dy, k>2 

Jmax{a;,s*_-^) y 

starting with G^^\x). Then, for G^'^^^^x) and sl^i given, we obtain si as the unique root 
X G (0, sl_i) of G'*^^^(x) = 0, or equivalently, from (l53l) . 



Sfc = exp <^ - 1 



/•/3 

:i+ln/3)2 + 2/ -^dy 



Similarly to the development of Section 5, we obtain the following result which gives a 
generalized version of formula ([1]). 

Lemma 3.2 Under (E^, si satisfies the recursion 



Sk = exp 



i=l 



{k-i + 2)\ 
where B^^i = (In s*)'''' — (In 

Let a = 1 + In /3. Then from Lemma 13.21 we can calculate the si successively as 



=exp{-(l + a)}. 



^2 = exp <^ — I 1 + ad 1 + -a 



S3 = exp 



l + aWl + ^a{l + (l + ^a)3/2} 



For m > 1, let gj^ be the scaled expected net payoff for the m-choice duration problem 
when n tends to infinity. Then we have the following result. 
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Lemma 3.3 Under (f5^). we have for m > 1 that 

/ m 

= - II In + mc 

\fc=i 

Proof. Similar to that of Lemma 12.61 
Table 1 



The asymptotic critical number sj^ for some values of m and c. 



c 




it* 








■^10 




0.0 


LOOOO 


0.1353 


0.0799 


0.0493 


0.0199 


0.0024 


0.0000 


0.1 


0.8942 


0.1513 


0.0990 


0.0698 


0.0416 


0.0281 


0.0280 


0.2 


0.7717 


0.1754 


0.1294 


0.1047 


0.0839 


0.0787 


0.0787 


0.3 


0.6130 


0.2208 


0.1898 


0.1761 


0.1690 


0.1684 


0.1684 



Table 2 

The asymptotic expected net payoff for some values of m and c. 



c 


Ql 


Q*2 


93 


95 


9i*o 


Q*oo 


0.0 


0.2707 


0.4725 


0.6208 


0.8066 


0.9656 


1.0000 


0.1 


0.1858 


0.3147 


0.4005 


0.4871 


0.5195 


0.5197 


0.2 


0.1053 


0.1700 


0.2062 


0.2322 


0.2363 


0.2363 


0.3 


0.0335 


0.0489 


0.0547 


0.0569 


0.0570 


0.0570 



Table [T] presents numerical values of (3 and for some values of m and c. Let /3' be 
the unique root x G (0, e~^] of — xlnx = c. It is intuitively clear that, as m — > oo, 
converges to /5' (= s^) because there is no benefit in choosing a candidate prior to 

Table [2] presents numerical values of for some values of m and c. It is interesting to 
compare, for example, g* = 0.0335 for c = 0.3 to g* = 0.2707 for c = 0, which implies 
that we can still gain positive expected payoff even when the acquisition cost is larger 
than the mean maximum payoff attainable when the acquisition cost is zero. This is not a 
contradiction. The stopping region shrinks as c gets large (see Table [1]) and positive mean 
payoff is assured by restricting our choice to a really good object. Table [2] suggests also 
that, as m ^ oo, converges to a value q^. This is given in the following lemma. 

Lemma 3.4 

C = (54) 

Proof. [231 As the arrival times of the n objects, we consider time epochs l/n,2/n, . . . ,n/n 
instead of 1, 2, . . . , n. When n oo, the transition probability p{i/n, j/n) = — 1)) 

then converges to the transition density p{x, y) = x/y'^ as i/n x, j/n — > y (see (|2])) 
and the candidates appear according to a non-homogeneous Poisson process with intensity 
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function A(x) = 1/x from (a), (b), in Section[2l That is, if we let N{a, b) denote the number 
of candidates that appear in time interval (a, b), th en N(a, b) becomes a Poisson random 
variable with parameter ln(6/a) (see Theorem 1 of [Gilbert and Mostellerl (119661 )). 



Let T{x) denote the time of the first candidate after time x if there is one and 1 if there 
is not. From the above T{x) has density fT{x){t) = p{x, t) = x/t^ on the time interval 
(x, 1) and probability mass x at 1. As the number of choices m oo, the optimal strategy 
chooses all the candidates that appear in time interval (/?, /?'). Thus the total proportional 
duration D is expressed as 



D 



T(/3)-T(/5'), ifT{(3')<(3 
0, if T{f3') > (3 



It is readily verified that T{(3) and T{/3') are independent. Hence by conditioning on T{f3'), 

E[D] = E[T{f])-T{f3')\T{(3') < /3]P{T{f]') < /?} 

= E[T{f3) ]P{T{(3') <(3}- E[T(/3') | T{(3') < (3]P{T{(3') < /5} 

Thus the expected net payoff is 



E[D- cN{(3, P') ] = {(3 -(]')- cln 



which yields (E 



4 Duration problem with replacement costs 



In this section a constant cost d = d{n) > is incurred each time there is replacement, 
whether or not the new candidate is the one to end the candidature of the previously 
chosen candidate. For simplicity we consider only the case where M ~ Ms{n) and ignore 
acquisition costs. The objective is to maximize the expected net payoff, that is, the total 
time of possession of a relatively best object less any replacement costs incurred. The 
multiple-choice duration problem with a replacement cost may be considered as a marriage 
and divorce problem, interpreting the replacement cost as alimony. 
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4-1 The degenerate distribution of the number of objects 



We treat the m-choice duration problem with replacement cost d > 0. In the m-choice 
problem we are allowed to replace objects up to m — 1 times, m > 2. We define the state 
of the process as in Section 3 and 'Wi''\ Uj;''^ and V}'''' similarly to in Section 6. 

Consider a candidate other than the first arriving at time i. As in Section 6, we may argue 
that such a candidate is never chosen unless 

d < UK{n)- (55) 
Further, [/f^^ — > if and only if a(n, d) <i < b{n, d). 

Once the first choice is made, the problem reduces to the (m — l)-choice problem with an 
acquisition cost d. Thus the main concern is to determine when to make the first choice. 
The optimal strategy can be summarized as follows. 

Theorem 4.1 For the m-choice duration problem with replacement cost condition / f53]) . 
there exists a sequence {si{d), S2{d), . . . , Sm.~i{d),tm.{d)) of integral critical numbers such 
that the optimal strategy first selects the first candidate (if any) to appear at or after time 
tm{d). Thereafter it replaces each previously chosen object with the first new candidate (if 
any) that appears at or after time Sk{d) but no later than b{n,d) if k more replacements 
are available (1 < k < m — 1), where b{n, d) = max{i : ul^"* > d}. 

Each Sk{d) is as in Theorem \3. 1\ while tm{d) < Sm{d) and is determined by 

tra{d) = min [i < bin, d) : Gf"\d) + g{d) > o} . (56) 



Proof. The part of the result relating to choices when fewer than m replacements are 
to be made is immediate from Theorem 14.11 so it remains to address the first choice of a 
candidate. 



As before 



and 



(m-l) 



vf-'\ ifj<s^„i(rf) 

ifj>.^_l(rf) 



The principle of optimality provides 



(m) _ ..(1) 



b{n,d) 



j=i+i 



(m-l) 



b{n,d) 

E 

j=i+i 



b(n,d) 



vr+ E p(^'j)v, 

j=i+l 



(m-l) 
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so that 



b(n,d) 
j=max{i+l,s„_i(d)) 

tG['\0)-zG\'\d) + tGt\d). 



Thus 



(m) 



^ = Gt\d)+g{d). 



Since by Corollary 13.11 a(d) > 0, (ISBl) implies that tm{d) < Sm{d). Further, ul^'^ > V, 
if and only if i > tm{d). Thus if the choice of a candidate appearing at time i < b{n, d) is 
preferable to that of the next candidate (if any) before time b{n,d), then it is preferable 
to the choice of any subsequent candidate. This concludes the proof. 



3(m) 



4-2 Asymptotics for the duration problem with replacement costs 



As n — i> oo, the cost condition fl55D is reduced to 



d = lim d{n)/n < e ^. 



(57) 



Let 5 = ^irn^6/?7,. Then under condition flFTI) . 6 is the unique root x G [e ^, 1) of —x In x = 
d. We have the following result concerning the limiting values = lim Sk/n {k > 1) and 



t*^ = lim tm/n. 



Lemma 4.1 Under (5^ , t*^ may be expressed in terms of s*^ as 



C = exp 



- 1 



;i + lns;;,)2- (2 + ln(5)ln(5 



(58) 



where (1 < k < m) satisfies the recursion 



Sk = exp 



i=l 



{k-t + 2)\ 
with Bk,t = (Ins*)^-' - {\n6f-\ 

Proof. Equation fl59|) is evident from Lemma [3. 2[ while fl58l) is immediate from fl56|) . 



(59) 



Let A = 1 + In (5. Then from fl58l) and fl59l) we have 
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\ 



For m > 2, let rj^ be the expected net payoff for tlie m-clioice duration problem when n 
tends to infinity. Then we have the following. 

Lemma 4.2 (i) If d> , then = 2e~^. 



(ii) If d < e ^ then = - 
Proof. The proof is omitted 



ET={st\nsl + t*^\nt*^ + {m-l)d 



Table 3 

The asymptotic critical number for some values of m and d. 



d 


H 


f* 
H 


f* 


''10 


t* 


0.1 
0.2 
0.3 


0.0916 
0.1063 
0.1243 


0.0656 
0.0885 
0.1186 


0.0397 
0.0725 
0.1154 


0.0270 
0.0684 
0.1151 


0.0268 
0.0684 
0.1151 



Table 4 

The asymptotic expected net payoff for some values of m and d. 



d 


r*2 


^3 


^5 


^1*0 


r* 

oo 


d 


rl 


^3 


* 

^5 


* 

^10 


r 


0.1 
0.2 
0.3 


0.4047 
0.3435 
0.2927 


0.4934 
0.3845 
0.3017 


0.5828 
0.4146 
0.3056 


0.6166 
0.4198 
0.3059 


0.6168 
0.4198 
0.3059 



Tables |3] and m give numerical values of t*^ and for some values of m and d respectively. 
The values of s*^ are given in Table [H if c is interpreted as d. Tables [3] and H] suggest that, 
as m — > oo, and converge to limits t*^ and respectively. The following lemma 
specifies these. 



Lemma 4.3 Let 6' be the unique root x G (0, e ^] of — xlnx = d for d < e ^ . Then 
f d^\ 



(60) 



where 



exp 



> . 



(61) 



26 



Proof. Relation (!6T]) is immediate from (!58l) . while (l60l) is immediate from Lemmas 13. 3[ 
[33] and 112] (ii). 



5 Final remarks 



he c l osely- re lated mu l tiple-ch o ice secretary p roblems have be en considered by iNikolaev 



197fi . ll977h . iTamakil fll979ah. iMoril fll984J). IStadiel fll98,5l). ISakaguchil (Il987l ). IWilson 



\ HHHH 1 ■^=|riHHdA7 h^^iiaa ■ \i • V 7 -k^H^Bia^d 1| | T " ^ ^ ' . ' *i ' \i 1/ j , j-, , \i 1/ j , . . , 

(119911 ). lAnd (119891 ) and others. iPreated (119941 ) gives some interesting results and a review 



of the literature. There are also results for Dynkin game models of the secretary problem 
when one player has t he opportunity to stop and ac cept a candidate more than once (see 
Szajowskil (120021 ) and lYasuda and Szajowskil (120021 )). 



Multiple-stopping mode ls have recently b een applied as a modelling tool for technical and 



economic phenomena. ISzajowskil (ll996l) has investigate d the double-disorder problem 



for di screte-time Markov processes. lAssaf et al.l (120041 ) and iKiihne and Riischendori 



(I2OO2I ) have considered the asymptotic prope rties of the double s topping procedure for iid 
rando m variables with known distribution. lAssaf et al.l (120021 ): lAssaf and Samuel-Cahn 
(120001 ) studied prophet inequalities in the case when the mortal has several choices. 



The model presented for the multiple-exchange duration problem is important for both 
applications and theoretical investigation of the optimization techniques in a stochastic 
envi ronment. There is opportuni t y for further extensions of the model in many directions 
(cf iMazalov and Tamakil (|200.i l2006h ). 



Some analytical aids exist for numerical and theoretical work, though we have not needed 
to invoke them in this study. We note in particular that the harmonic number i/„ can be 
expressed an alytically as Hn = 'y + il^oln + 1) , where 7 = 0.577216 is the Euler-Mascheroni 



constant (see 



Gradshteyn and Ryzhikl (I2OOOI )) and the digamma function xpo is defined by 



— InT z = -f^ 

dz ^ ' T{z) 



see 



Abramowitz and StegunI (I2OOOI ) p. 260). 
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